[Remote Store] Sync segments in refresh listener on refresh after commit #10830

ashking94 · 2023-10-22T08:54:52Z

Description

With this PR, we are doing following changes -

On replica promotion, we update the segments tracker with remote uploaded files on remoteDirectory.init() method invocation in runAfterRefreshExactlyOnce(..). This ensures that we do not see spikes in refresh lag bytes on failovers.
There is a case during failover where the runAfterRefreshExactlyOnce gets executed, but the syncSegments does not. This happens due to the this.primaryTerm != indexShard.getOperationPrimaryTerm() evaluating true for the first time, but evaluating to false on the second time.

Related Issues

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-10-22T09:26:42Z

Compatibility status:

Checks if related components are compatible with change 17b9dcd

Incompatible components

Incompatible components: [https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

github-actions · 2023-10-22T09:37:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28720/
CommitID: 71e4371
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-22T09:40:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/28718/
CommitID: b916e0f

codecov · 2023-10-22T09:41:30Z

Codecov Report

Merging #10830 (17b9dcd) into main (51626d0) will decrease coverage by 0.03%.
Report is 9 commits behind head on main.
The diff coverage is 90.04%.

@@             Coverage Diff              @@
##               main   #10830      +/-   ##
============================================
- Coverage     71.31%   71.29%   -0.03%     
- Complexity    58671    58728      +57     
============================================
  Files          4860     4869       +9     
  Lines        276335   276490     +155     
  Branches      40198    40204       +6     
============================================
+ Hits         197068   197123      +55     
- Misses        62803    62886      +83     
- Partials      16464    16481      +17

Files	Coverage Δ
...upport/replication/TransportReplicationAction.java	`77.10% <100.00%> (-3.58%)`	⬇️
...ava/org/opensearch/cluster/node/DiscoveryNode.java	`91.62% <100.00%> (+0.17%)`	⬆️
...a/org/opensearch/common/network/NetworkModule.java	`92.20% <100.00%> (+0.20%)`	⬆️
...rg/opensearch/common/settings/ClusterSettings.java	`92.85% <ø> (ø)`
.../java/org/opensearch/gateway/GatewayMetaState.java	`69.56% <100.00%> (+1.04%)`	⬆️
...earch/index/remote/RemoteStorePressureService.java	`96.61% <ø> (-3.39%)`	⬇️
...ch/index/remote/RemoteTranslogTransferTracker.java	`80.97% <100.00%> (+2.06%)`	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	`70.56% <100.00%> (+0.81%)`	⬆️
...rg/opensearch/index/translog/RemoteFsTranslog.java	`75.12% <100.00%> (+0.62%)`	⬆️
server/src/main/java/org/opensearch/node/Node.java	`85.31% <100.00%> (+0.09%)`	⬆️
... and 17 more

... and 457 files with indirect coverage changes

ashking94 · 2023-10-22T11:16:34Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/28720/

CommitID: 71e4371
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Flaky test - #2775

github-actions · 2023-10-22T15:39:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationMultiSearchDuringFetchPhase

URL: https://build.ci.opensearch.org/job/gradle-check/28737/
CommitID: 7f7a5cd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-10-22T15:44:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28738/
CommitID: 83519fd
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

github-actions · 2023-10-23T05:42:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: TIMEOUT ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28753/
CommitID: e130ba9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreStatsIT.java

test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java

Signed-off-by: Ashish Singh <[email protected]>

github-actions · 2023-10-23T08:01:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

URL: https://build.ci.opensearch.org/job/gradle-check/28780/
CommitID: 17b9dcd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-10-23T08:10:46Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌
URL: https://build.ci.opensearch.org/job/gradle-check/28772/
CommitID: 260f783
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

ashking94 · 2023-10-23T08:37:30Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ABORTED ❌

URL: https://build.ci.opensearch.org/job/gradle-check/28772/

CommitID: 260f783
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

This is a build failure before the rebase of BWC PR that was causing this issue.

ashking94 · 2023-10-23T08:38:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕

TEST FAILURES:
      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled
URL: https://build.ci.opensearch.org/job/gradle-check/28780/

CommitID: 17b9dcd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Flaky test - #10755.

…mit (#10830) * [Remote Store] Sync segments in refresh listener on refresh after commit Signed-off-by: Ashish Singh <[email protected]> * Add Integration Tests Signed-off-by: Ashish Singh <[email protected]> * Add comments and java doc Signed-off-by: Ashish Singh <[email protected]> --------- Signed-off-by: Ashish Singh <[email protected]> (cherry picked from commit 7453daa) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…mit (#10830) (#10849) * [Remote Store] Sync segments in refresh listener on refresh after commit * Add Integration Tests * Add comments and java doc --------- (cherry picked from commit 7453daa) Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…mit (opensearch-project#10830) * [Remote Store] Sync segments in refresh listener on refresh after commit Signed-off-by: Ashish Singh <[email protected]> * Add Integration Tests Signed-off-by: Ashish Singh <[email protected]> * Add comments and java doc Signed-off-by: Ashish Singh <[email protected]> --------- Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

github-actions bot added enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Storage:Remote v2.12.0 Issues and PRs related to version 2.12.0 labels Oct 22, 2023

ashking94 added the skip-changelog label Oct 22, 2023

ashking94 force-pushed the 10821 branch from b916e0f to 71e4371 Compare October 22, 2023 09:06

ashking94 marked this pull request as ready for review October 22, 2023 11:10

ashking94 requested review from abbashus, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, peternied, reta, Rishikesh1159 and ryanbogan as code owners October 22, 2023 11:10

ashking94 requested review from shwetathareja, sohami and VachaShah as code owners October 22, 2023 11:10

ashking94 mentioned this pull request Oct 22, 2023

[BUG] test failure ':sandbox:plugins:concurrent-search:test' QueryProfilePhaseTests.testTerminateAfterEarlyTermination #2775

Closed

ashking94 force-pushed the 10821 branch from 7f7a5cd to 83519fd Compare October 22, 2023 15:04

gbbafna reviewed Oct 23, 2023

View reviewed changes

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java Show resolved Hide resolved

ashking94 mentioned this pull request Oct 23, 2023

[BUG][Flaky Test] org.opensearch.remotestore.RemoteStoreStatsIT.testStatsOnRemoteStoreRestore #10831

Closed

ashking94 force-pushed the 10821 branch from e130ba9 to 260f783 Compare October 23, 2023 06:07

ashking94 commented Oct 23, 2023

View reviewed changes

server/src/internalClusterTest/java/org/opensearch/remotestore/RemoteStoreStatsIT.java Outdated Show resolved Hide resolved

ashking94 commented Oct 23, 2023

View reviewed changes

test/framework/src/main/java/org/opensearch/test/OpenSearchIntegTestCase.java Outdated Show resolved Hide resolved

sachinpkale approved these changes Oct 23, 2023

View reviewed changes

ashking94 added 3 commits October 23, 2023 12:48

[Remote Store] Sync segments in refresh listener on refresh after commit

ad6c7e3

Signed-off-by: Ashish Singh <[email protected]>

Add Integration Tests

1da1727

Signed-off-by: Ashish Singh <[email protected]>

Add comments and java doc

17b9dcd

Signed-off-by: Ashish Singh <[email protected]>

ashking94 force-pushed the 10821 branch from 260f783 to 17b9dcd Compare October 23, 2023 07:19

ashking94 mentioned this pull request Oct 23, 2023

[BUG] org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled is flaky #10755

Open

sachinpkale merged commit 7453daa into opensearch-project:main Oct 23, 2023
16 checks passed

sachinpkale added the backport 2.x Backport to 2.x branch label Oct 23, 2023

opensearch-trigger-bot bot mentioned this pull request Oct 23, 2023

[Backport 2.x] [Remote Store] Sync segments in refresh listener on refresh after commit #10849

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Sync segments in refresh listener on refresh after commit #10830

[Remote Store] Sync segments in refresh listener on refresh after commit #10830

ashking94 commented Oct 22, 2023 •

edited

Loading

github-actions bot commented Oct 22, 2023 •

edited

Loading

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 22, 2023

codecov bot commented Oct 22, 2023 •

edited

Loading

ashking94 commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 22, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

github-actions bot commented Oct 23, 2023

ashking94 commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

[Remote Store] Sync segments in refresh listener on refresh after commit #10830

[Remote Store] Sync segments in refresh listener on refresh after commit #10830

Conversation

ashking94 commented Oct 22, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Oct 22, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 22, 2023 • edited Loading

Codecov Report

ashking94 commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 22, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 commented Oct 23, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 commented Oct 22, 2023 •

edited

Loading

github-actions bot commented Oct 22, 2023 •

edited

Loading

codecov bot commented Oct 22, 2023 •

edited

Loading